Distance between two distributions

Total Variation Distance (TVD, or variation distance or statistical distance)

= the maximum absolute difference in probabilities assigned to any single event by the two distributions.

d_{TVD}(P, Q) = \frac{1}{2} \sum_x |P(x) - Q(x)|$$, where x ranges over all possible values in the sample space - range in [0, 1], where 1 = no events in common - symmetric - robust to outliers and can capture the largest discrepancy between discrete/continuous distributions ### Wasserstein Distance (Earth Mover's distance or Kantorovich-Rubinstein distance) = the minimum amount of work needed to transform one distribution into the other, where "work" is defined in terms of the cost of moving mass from one point to another. $$W_p(P, Q) = \left( \inf_{\gamma \in \Pi(P, Q)} \int_{\mathcal{X} \times \mathcal{X}} d(x, y)^p d\gamma(x, y) \right)^{\frac{1}{p}}

range in [0, inf/diameter of the support of the distributions]
useful for comparing distributions with complex shapes or high-dimensional data

Hellinger Distance

= the similarity between two probability distributions by comparing the square root of the difference between the square roots of the probability densities.

H^{2} (P, Q) = \frac{1}{2} \sum_{x} (\sqrt{P (x)} - \sqrt{Q (x)})^{2}

range in [0, 1]
sensitive to differences in the tails of distributions

Kullback-Leibler Divergence (KL Divergence)

see Information theory and Entropy in Neuroscience#Kullback-Leibler Divergence

range in [0, inf]
not symmetric

Jensen-Shannon Divergence

= a symmetrized version of KL divergence that measures the similarity between two distributions by averaging their KL divergences from the average of the two distributions.

D_{J S} (P | | Q) = \frac{1}{2} D_{K L} (P | | M) - \frac{1}{2} D_{K L} (Q | | M) $ $ w h e r e $ M = \frac{1}{2} (P + Q) $ i s t h e a v e r a g e o f t h e t w o d i s t r i b u t i o n s - r a n g e i n [0, l o g (2)] - s y m m e t r i c - c a p t u r e s b o t h t h e d i v e r g e n c e a n d c o n v e r g e n c e b e t w e e n d i s t r i b u t i o n s .